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This paper focuses on recursive estimation of time varying au- 
toregressive processes in a nonparametric setting. The stability of the 
model is revisited and uniform results are provided when the time- 
varying autoregressive parameters belong to appropriate smoothness 
classes. An adequate normalization for the correction term used in 
the recursive estimation procedure allows for very mild assumptions 
on the innovations distributions. The rate of convergence of the point- 
wise estimates is shown to be minimax in /3-Lipschitz classes for 
< P < 1. For 1 < j3 < 2, this property no longer holds. This can 
be seen by using an asymptotic expansion of the estimation error. A 
bias reduction method is then proposed for recovering the minimax 
rate. 

1. Introduction. Suppose that we have real- valued observations (X± n , 
X2, n , ■ ■ ■ ,Xn,n) from a time- varying autoregressive model (TVAR) 

d 

(1) X k ^ n = ^6 i ((k-l)/n)X k _ i , n + o-{k/n)e k , n , fc = l,...,n, 

i=l 

where 

" { e k,n}i<k<n is a triangular array of real-valued random variables referred 
to as the (normalized) innovations, 

- 0(t) := [#i (i) • • -9d(t)] T , t G [0, 1], is a d-dimensional vector referred to as 
the local autoregression vector, 

- o~(t), t 6 [0, 1], is a nonnegative number referred to as the local innovation 
standard deviation. 
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This recurrence equation may be more compactly written as 

(2) Xk t n = Ok-l,nX-k-l,n + ^k,n £ k,n, k=l,...,n, 

where 

^-k,n '■= [Xk,nXk-l,n ' ' ' X k 

—d+l,n\ i 

8 k , n :=0(k/n) = [6i{k/n)9 2 (k/n)---6 d (k/n)} T and a k , n = a(k/n). 

TVAR models have been used for modeling data whose spectral content 
varies along time (see, e.g., [12, 15, 22] for early references). TVAR models 
are also closely related to the general class of locally stationary processes 
(see [5, 6, 7] and Remark 3 below for definitions and properties). 

In this paper we focus on the estimation of the functions 1 1— ► G(t) [we 
leave aside a(t) for brevity] from the observations {Xo, n > X k ^k > 1} (here 
we add the initial conditions Xo, n in the observations set for convenience). 
This problem is reminiscent of nonparametric curve estimation on a fixed 
design, a problem which has received considerable attention in the literature. 
A natural approach consists of using a stationary method on short overlap- 
ping segments of the time series (see, e.g., [8]). An alternative approach, 
first investigated by [3] for first order TVAR models, consists of estimating 
the regression function recursively in time. More precisely, at a given time 
t £ (0,1), only observations that have been observed before time t are used 
in the definition of the estimator: n (t) = n (t, Xo, n , Xi >n , . . . , Xr nt i n ), where 
[x] denotes the integer part of x. This approach is useful when the obser- 
vations must be processed on line (see, e.g., [18, 19, 21]). We focus in this 
contribution on the normalized least square algorithm (NLMS), which is a 
specific example of a recursive identification algorithm, defined as follows: 

0o,n(AO :=0, 

(3) 

0fc+l,n(A*) : = 0k,n((l) + fJ>(Xk+l,n 

where k goes from zero to n — 1, fx is referred to as the step-size and | • | 
denotes the Euclidean norm. At each iteration of the algorithm, the param- 
eter estimates are updated by moving in the direction of the gradient of the 
instantaneous estimate (X^i^ — # T Xfc jn ) 2 of the local mean square error 
E[(A A fc + i n — T X^. in ) 2 ]. The normalization (1 + /i|Xfc in | 2 ) -1 is a safeguard 
again large values of the norm of the regression vector and allows for very 
mild assumptions on the innovations [see (Al) below] compared with the 
LMS, which typically requires much stronger assumptions (see [20]). Ex- 
tensions to a more sophisticated iterative rule, for example, the so-called 
recursive least-square (RLS) algorithm, are currently under investigation. 



^fc,n(^) X fc,n) 



1 + //|X/. J 
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We define a pointwise estimate of t \— > 6(t) as a simple interpolation of 
dk,n(n),k = l,...,n, that is, 

(4) 9 n (t;fi):=e [tn]t M, i€[0,l],n>l. 

Observe that, for all t E [0,1], O n (t;fj,) is a function of Xo, n) {Xi >n ,l = 
1, . . . , [tn]} and //. 

The paper is organized as follows. In Section 2 we introduce the assump- 
tions and state the main results of this paper, namely, uniform risk bounds 
for n , a lower bound on the minimax L 2 -risk and precise approximations 
of the risk for 6 n . We also discuss a technique to correct the bias of the 
estimator. In Section 3 the basic results used in the paper for dealing with 
weak dependence are presented. In Section 4 a four-step proof of the uniform 
upper bound for the L p -risk of 6 n is given. Section 5 then provides the proof 
of the minimax lower bound. In Section 6 further approximation results are 
given, from which we obtain the proofs of the risk approximations for n 
stated in Section 2. 

2. Main results. The space of m x n matrices is embedded with the 
operator norm associated to the Euclidean norm, which we denote by 

(5) \A\ := sup \Ax\. 

Observe that for a row or column vector, its Euclidean norm coincides with 
its operator norm. For any random variable Z in a normed space (Z, | • |), 
we denote by ||Z|| P := (E|Z| p ) 1 / p its L p -norm. Throughout the paper, it is 
assumed that, 

(Al) for all n > 1, the random variables {£fc, n }i<fc<n are independent, 
have zero mean and unit variance and are independent of the initial 
conditions Xo, n . In addition, sup n>1 ||Xo, n ||g < oo and e* := 

SUPi< fc < n \\Sk,n\\q < OO, 

where the moment order q > 2 will be set depending on the context. The 
triangular array of random variables {Xj. n , 1 < k < n} defined by (1) is 
parameterized by (0,a). To keep track of the dependence on (6, a), for all 
random variables Z defined as a function of these variables, we shall adopt 
the notation convention Ee j(J [Z] for the expectation of Z. In the case of a 
random element Z taking its values in the normed space (Z, | • |), its L p -norm 
will be denoted by 

l|Z|| p , , CT :=(E 0jCT |Zn 1/p . 

A classical problem in nonparametric estimation is to derive uniform bounds 
for the pointwise L p -risk ||0 n (£; /i) — 0(t)\\ P: e,a- for (6, a) in some appropriate 



4 



E. MOULINES, P. PRIOURET AND F. ROUEFF 



classes of functions. In the sequel we denote by 

d 

(6) 6(z;t):=l-J2^j(t)z j , z£C, 

i=i 

the local time-varying autoregressive polynomial associated with 6 at point t. 
The function classes that will be considered rely on two kinds of properties. 
First, the roots of the time- varying autoregressive polynomial associated 
with 6 are required to stay away from the unit disk. Second, as in function 
estimation from noisy data, 9 and a are supposed to be smooth in some 
appropriate sense. The first condition is unusual in the nonparametric func- 
tion estimation setting and deserves some elaboration. Let us recall some 
elementary facts from the theory of autoregressive processes. The process 
{Z k } is an AR(d) process with parameters 1? G M. d and q 2 > if {Z k } is 
second-order stationary and satisfies the difference equation 

d 

(7) Z k = ^2^Z k -j + e k , keZ, 

j=i 

where {£k}kez is a centered white noise with variance ? 2 . A sufficient and 
necessary condition for the existence of {Z k } is that the autoregressive poly- 
nomial z *— ► $(z) := 1 — J2j=i tyz'' does not vanish on the unit circle (see [4]). 
In this case, the stationary solution {Z k } to (7) is unique and there exists 
a sequence {tpk}, such that J2i < 00 an d Z k = J2i ipl^k-l f° r an k £ Z. 
Furthermore, the sequence {ipi} is causal, that is, ipi = for all I < 0, if and 
only if the function z \— > $(z) does not vanish on the disk \z\ < p~ 1 , for some 
p < 1. This motivates the following definitions in the time varying setting. 
For p > 0, we denote 

(8) S{p) := {6 : [0, 1] -> R d , 6(z; t) £ for all \z\ < p~ l and t G [0, 1]}. 

Concerning the smoothness condition, different classes of functions can 
be considered. In the original paper by [6], it is assumed that the functions 
1 1 — > 6(t) and t >— > a(t) are Lipschitzian. In this paper we consider a wider 
range of smoothness classes which are now introduced. For any (3 G (0,1], 
denote the /3-Lipschitz semi-norm of a mapping f : [0, 1] i — ^ R l by 

|f(t)-f(s)| 

f \A,f3 = SUp — -3—. 

t^s \t-s\P 

And define for < L < oo the /3-Lipschitz ball 

(9) Ai(/3 t L):={f:[0,i\-^R l t \f\ Al p<L, sup |f(t)|<L 

L te[o,i] 
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For all (3 > 1, the /3-Lipschitz balls are classically generalized as follows. Let 
fcGN and a G (0, 1] be uniquely denned by (3 = k + a. Then we define 

(10) A,(ftL):={f:[0,l]^R l ,|fW| Aa <L, sup |f(t)|<A 

I- te[o,i] J 

where f( fc ) is the derivative of order k. 

For all j3 > 0, L > 0, < p < 1, and < <r_ < ct + < oo, we define 

C(/3, L, p, a-, a+) := {(9, a):9 £ A d (f3, L) n a : [0, 1] - [a^,a + ]}. 

We will simply write C whenever no confusion is possible. It is interesting to 
observe that, for particular choices of L and p, C reduces to the more con- 
ventional smoothness class {(6,a):0 G A^(/3, L),a: [0, 1] — ► [<7_,<r+]}. This 
follows from the following lemma. 

Lemma 1. For all positive p, we have 

(11) i?(l/ v /p- 2 + --- + p- M ) C C B((l + p) d - 1), 
where B(a) is the sup-norm ball {6 : [0, 1] — > M d , sup tG [ ,i] 1^(^)1 — a l- 

Proof. Note that 6»(t; 0) = 1 for all t G [0,1]. Let Ai(t), . . . , \ d {t) be the 
reciprocals of the zeros of the polynomial z \— > 0(z;t). Hence, 9 G S(p) iff, 
for all t G [0, 1] and k = 1, . . . , d, |A fc (t)| < p. For all i G [0, 1] and k = 1, . . . , d, 
using the Cauchy-Schwarz inequality, we have 



1 



8=1 



< 



1/2 



-2i 

.1=1 / 



Hence, the first inclusion in (11). We further have 

d 

9(z;t)=l[(l-X k (t)z). 
k=i 

The coefficients of 9 are then given by 

(12) 6 k (t) = (-l) k J2 A n (t)---A lfe (t), k = l,...,d. 

l<ii<— <ik<d 

A simple computation then gives the second inclusion in (11). □ 

Remark 1. In the case d = 1 considered in [3], S(p) = B(p). 

We are now in a position where we can state the main results of this 
paper. We first provide a uniform upper bound on the pointwise L p -risk of 
the NLMS estimator. 
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Theorem 2. Assume (Al) with q>4 and let p G [l,q/3). Let (3 G (0, 1], 
L > 0, < p < 1 and < o~_ < <r+. T/ien i/iere exist M, 5 > and /zo > 
snc/i that, for all p G (0, po], n > 1, t G (0, 1] and (0, cr) G C(/3, L, p, cr_, <r+), 

(13) ||0 n (i; /i) - < M(|0(O)|(1 - 5p) tn + Jp + (n»)-P). 

Corollary 3. For all rj G (0,1) and a > 0, there exists M > snc/i 
i/iafj /or aZZ (0, a) G C(/3, L, p, o\_, <r + ), 

sup ||0 n [t;an- 2 ^( 1+2 «] - 0(t)|L < M^ 1+2 « 

The upper bound in (13) has three terms. Anticipating what will be said 
in the proof section (Section 4), the first term |0(O)|(1 — 5p,) tn reflects the 
forgetting of the initial error of the NLMS estimator. The second term is the 
so-called lag-noise term, which accounts for the fluctuation of the recursive 
estimator which would occur even if 1 1— > 6(t) were constant. The third term 
controls the error involved by time evolution of 6(t) and mainly relies on the 
smoothness exponent (3. Corollary 3 is then obtained by choosing the step- 
size in order to minimize this upper bound. Observing that the first term 
is negligible for t>rj>0 and balancing the two remaining ones yields p, oc 
n 2 ^/(i+2/3)_ Thjg corollary says that, for all j3 G (0, 1], under the /3-Lipschitz 
assumption, for t G (0, 1] the L p -risk of the NLMS estimator at point t has 
rate n~/ 3 /( 1 + 2 / 3 ). 

We now provide a lower bound on the L 2 -risk for any estimator S n of 
9{t) computed from observations Xo, n > X\ n , . . . , X nn . Let us stress that this 
lower bound is not restricted to recursive estimators, that is, we do not 
require 8 n to depend only on Xo, m Xi <n , . . . ,Xr nt i n . Denote by 

MSEM e , a (dn,t) := E 0>tT [(S n - 0(t))(6 n - 0(t)f] 

the mean square error matrix (MSEM) at t G [0, 1]. Consider the following 
assumption: 

(A2) For all n G N and 1 < k < n, e^ n has an absolutely continuous density 
Pk n w.r.t. the Lebesgue measure whose derivative p\. n satisfies 

2i 



E 



Pk,n, 
Pk,n 



and X e := sup E 

Kk<n 



Pk,n I i 

K £ k,n, 

Pk,n 



< OO. 



We have the following result, the proof of which appears in Section 5. 

Theorem 4. Assume (Al) with q = 2 and (A2). Let (3>0, L>0, p G 
(0, 1), < cr_ < a + . Then there exists a > such that, for all n > 1, t G [0, 1], 
and for all estimators 8 n := 5 n (Xo,n, X\. n , . . . , X n>n ) G M. d , 

(14) inf sup u T MSEM 0ja (S n ,t)u>an- 2l3/{1+2/3 \ 

\ u \ =1 {0,cr)eC 
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where C := C((3, L, p, cr_, <r+). 

Remark 2. Note that u T MSEM 0i(T (5 n ,i)u is the mean square error of 
u T S n for estimating u T 6(t). 

Corollary 3 and Theorem 4 show that, under (Al) with q > 6 and (A2), 
the IP error rate is minimax for p>2 within the class C(j3, L, p, cr_, cr+) if 
/? G (0, 1] . The question arises whether the upper bound derived in Theorem 2 
generalizes for /3 > 1 in such a way that, as in Corollary 3, for an appropriate 
step-size p(n), 6 n (t;p(n)) achieves the rate of the lower bound derived in 
Theorem 4. It turns out that this is not the case, except in a very particular 
situation, namely, when 9 is the constant function. This may be shown by 
using precise approximations of the risk, completing the upper bound given 
in Theorem 2. Such approximations primarily rely on the fact that, as n tend 
to infinity, the local sample X& n of the TVAR process approximately has the 
same second-order statistics as the stationary AR(d) process with parameter 
{6{k/n),a{k/n)). In the following we provide a precise statement of this fact 
and then state the approximations of the risk. For this purpose, we need to 
introduce further notation. For (3 > 0, L > 0, p G (0, 1) and < cr_ < a + , we 
let 

(15) C*G0, L, p, *-,*+) := {(6, a) G C(f3, L, p, a^,a + ):a€ A a a+)}. 

We use the shorthand notation C* when no confusion is possible. The obvious 
relation 

C(P,L,p,a^,a + ) ^C*(P,L,p,a_,a + ) 5 C(/3, L, p,a + ,a + ) 

implies that Theorem 2 and Theorem 4 are still valid when replacing C by C* . 
Following the formula for spectral densities of stationary AR(d) processes, 
we respectively denote 

(16) f(X;t,0,a) := ^\6(e a ;t)\- 2 , -vr < A < vr, 

(17) [E(i, 0, a)] Kl := f e lX ^f(X; t, 6, a) dX, l<k,l< d, 

the local spectral density function and the local d- dimensional covariance ma- 
trix associated with (6, a) at point t. The covariance matrix E0 o .[Xfc n XjT ] 
can be approximated by the local covariance matrix at point k/n as follows. 

Proposition 5. Assume (Al) with q>2. Let /3 G (0, 1], L > 0, < p < 
t < I, and < <r_ < 0"+ . Then there exists M > such that, for all 1 <k <n 
and (6, a) G C*((3, L, p, a_,a + ), 

|E 0)CT [X fe , n Xjj - E(k/n,0,a)\ < M(r fc |E[X , n X^ n ] - S(0, 0, a)\ + n^). 
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Remark 3. This approximation result can serve as an illustration of 
how the TVAR process fits into the locally stationary setting introduced 
by Dahlhaus [5]. Observe that if j3 = 1 and E[Xo, n Xo n ] = £(0,0, cr), the 
rate for the approximation error between the local sample covariance ma- 
trix Ee CT [Xfc n XjT ] and its local stationary approximation £(&/n, 0,cr) is 
n , which coincides with the approximation rate required in the locally 
stationary setting introduced in [5]. Precise conditions upon which a TVAR 
process is locally stationary are given in [6]. 

We obtain the following computational approximation of the pointwise 
MSEM for n . 



Theorem 6. Assume (Al) with q > 11. Let (3 e (0, 1], L > 0, p < 1, and 
< (j_ < cj+ and let (0, a) G C*(/3,L, p,o~-,o~+). Let t S (0,1] and assume 
that there exists Qt.fi €= L' > and (3' > (3 such that, for all u 6 [0,t], 

(18) \6(u) - 0(f) - O t ,p{t - uf\ < L'(t - uf. 

Then there exist M > and Ho> such that, for all p G (0,/io] an d n>l, 



(19) 



MSEM 0jCT (e n (t; ,1) - T( f + V v-P(t, 0, v)0 t , 



(pnf " ^-/^PJ r 2 



1/2 



< M(v^(v^+ (pn)^/ 2 ) + (jMnr^fjxn)-^ + (/in)^' + ^)), 
where T is the usual Gamma function and I is the dx d identity matrix. 

Remark 4. Let a£R. The a-fractional power of a diagonal matrix D 
with positive diagonal entries is the diagonal matrix D a obtained by raising 
the diagonal entries to the power a. The a-fractional power of a symmetric 
positive-definite matrix A = UDU T , where U is unitary and D is diagonal, 
is then defined by A a = UD a U T . 

Using (19), as (p + (/m) _1 ) — > 0, we have the following asymptotic ap- 
proximation of the MSEM: 

MSEM 0i(T (0 n (f ; p) - T(f3 + l){pn)-^(t, 0, a)0 t>p ) = + 

If /3 < 1, this gives the leading term of an asymptotic expansion of the 
MSEM, which allows one to compare the performance of 6 n with other 
estimators achieving the minimax rate. The deterministic correction T(f3 + 
l)( / un) _ ^£"' 3 (t, 0, cr)0£ j( g can be interpreted as the main term of the bias and 

the term p a ^ I as the main term of the covariance matrix. 
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Remark 5. An essential ingredient for proving Theorem 6 is to ap- 
proximate the expectation of the term X^ n Xfe jn /(l + p|Xfc jn | 2 ) appear- 
ing in (3). Roughly speaking, for small enough p, a good approximation 
is Ee )0 .[XjjT n Xfc in ], which itself is well approximated by £(/c/n, 6, a) by using 
Proposition 5. If one replaces the normalization factor 1 + /i|Xfc n | 2 by the 
more classical (in the stochastic tracking literature) 1 + |Xfc n | 2 , the compu- 
tation of the deterministic approximation would be much more involved as 
the normalization does not reduce to one as p tend to zero. 

Remark 6. Equation (18) is, for instance, valid if 6 behaves as a sum of 
nonentire power laws at the left of point t, say, 9{u) = ^t,i/2V / * — u + 0(t — u). 
Another simple case consists in assuming that (6, a) G C*(0,L, p,a-,a + ) 
for > 1. Then (18) is obtained with = 1 and 6 t) p = —0(t) by using 
a first-order Taylor expansion. Hence, in this case, the main terms of the 
MSEM are of order p + (pn)~ 2 unless 6(t) = 0, in which case the deter- 
ministic correction in the MSEM vanishes. This implies that the estimator 
9 n (t; p) cannot achieve the minimax rate obtained in Theorem 4 in the class 
(0,o) G C*(0 , L, p, <t_,(t+) for > 1 unless 6 is a constant function. 

If the smoothness exponent belongs to (1,2], the following result applies. 

Theorem 7. Assume (Al) with q > 4 and let p G [1,^/4). Let j3 G (1,2], 
L > 0, p G (0, 1), and < cr_ < a + . Then, for all rj G (0, 1), there exist M > 
and no > such that, for all (6, a) G C*(f3, L, p, cr_, <r+), t G [rj, 1], n > 1 and 

pe (o,po], 

\\e n (t; p) - 9{t) + (pn)- l ^-\t, 6, u)e{t)\\ p>e ,a < M{Jp+{pn)- p + {pn)~ 2 ). 

Applying a technique inspired by the so-called Romberg method in numer- 
ical analysis (see, e.g., [2]), we are now able to propose a recursive estimator 
which achieves the minimax rates for /3 G (1,2]. This estimator is obtained 
by combining the recursive estimators 0(t; •) associated to two different step- 
sizes. More precisely, let 

Q n {t;n,i) := ip n {t;p) - jG n (t;jp,)), 

1-7 

where 7 G (0, 1). We obtain the following result. 

Theorem 8. Assume (Al) with q > 4 and let p G [l,q/4). Let j3 G (1,2], 
L > 0, p < 1, and < er_ < cr + . For a// 77 G (0, 1), there exist M > and 
po > such that, for all £ (0,1) , (6, a) G C*(/3, L, p,a-,a + ) ,n>l and 
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H G (0,fi o ], 

sup \\6 n (t;p,-f) - 0(t)\\ Pj g >a 
te[vA] 

(20) 

PROOF. Let rj G (0, 1), 7 G (0, 1) and t G [77, 1]. One easily checks that 

e n {t\ii,i)-e{t) 

= (1 - 7)" 1 (^n(i; M) " »(t) + (/xn)-^- 1 ^, 0, (7)©(t) 

- 7 (0 n (t; 7/i) - 9{t) + ( 7 /in)- 1 S- 1 (t, 0,cr)0(t))). 

The Minkowski inequality, Theorem 7 and the bounds y^y < 7 _/3 < 7~ 2 
yield (20). □ 

Corollary 9. For aZZ 7,77 G (0,1), /3 G (0,2] and a > 0, i/iere exisis 
M > s-uc/i £/ia£, /or a// (0,ct) e C*(f3,L,p,a-,a + ), 

sup ||0„(t;an-W+2/?) )7) _ 0( t )|| < M^n^/f 1 ^. 

3. Exponential stability of inhomogeneous difference equations. Let us 

consider a sequence {Z&, > 0} of random vectors satisfying the inhomoge- 
neous difference equation 

(21) Z k = A k Z k _ 1 + B k \J k , k>l, 

where {U k ,k > 1} is a sequence of independent random vectors and A = 
{A k ,k > 1},B = {B k ,k > 1} are two sequences of deterministic matrices 
with appropriate dimensions. The pair (A, B) is said to be exponentially 
stable if there exist constants C > and p G (0, 1) such that 



(22) sup 

fc>0 



k+m 

n * 

l=k+l 



<Cp m for all m, >0 and B* : = sup \B k \ < 00, 

fe>i 



with the convention that Yl^™ Ai := A k+m A k+m _i ■ ■ • A k . We clearly have 
the following: 

Proposition 10. Let p G [l,oo]. Suppose that (A,B) is exponentially 
stable. Then there exists a positive constant M depending only on C, p and 
B* such that 

||Z fc ||p<Af(U*p+||Z y, ken, 
where XJ*p : = sup fc>1 ||Ufc|| p . 
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Exponential stability implies exponential forgetting of the initial condition 
in the following setting. Let (E, | • |e) and (F, | • |p) be two normed spaces, 
let m be a positive integer and p a nonnegative real number. We denote by 
Li(E, m, F;p) the linear space of mappings <fi : E m — ► F for which there exists 
Ai > such that, for all (x±, . . . ,x m ) and (yi, . . . ,y m ) G E m , 

\<t>(x\,...,x m ) -4>(yi,-.-,ym)\F 

(23) < Ai(|xi - 2/1 1 E H h|x m -y m | E ) 

x (1 + + Me H h \x m \l + \y m \l). 

This implies that there exists A2 > such that, for all (xi, . . . ,x m ) G E m , 

(24) |0(xi, . . . ,x m )\ F < A 2 (l + |x!^ +1 + • • • + |x m |P +1 ). 

Denote by |</>|Li( P ) the smallest A satisfying (23) and (24). Li(E, m, F;p) is 
called the linear space of p- weighted Lipschitz mappings and the 
p-weighted Lipschitz norm of (j). We now state the exponential forgetting 
property. In the sequel, for any integrable r.v. Z, E^[Z] denotes the condi- 
tional expectation of Z given the cr-field J- and inequalities involving random 
variables will be meant in the almost sure sense. 

Proposition 11. Let p> 0. Assume that U* +1 is finite and that (22) 
is satisfied for some C > and p < 1. Let <j) G Li(]R rf , m, M.;p), where m is a 
positive integer. For all k>0, let us denote T k := <j(Zo,U/, 1 <l <k). Then 
there exist constants C\ and C 2 (depending only on C, p, U* +1 ,.B* and p) 
such that, for all < I < k and = j\ < ■ ■ ■ < j m , 

|E^[* fc+ r]|<C 1 m|<A| Li(p) (l + / (p+1) |Z fc r 1 ), 

(25) |E^[* fc+r ] -E*i[* fc+r ]| < C 2 m/ 1 0| Li(p) (1 + I Z fc |f +1 +E^ I Z fc |P +1 ), 

|E^[* fc+ r] - E[$ fe+r ]| < C 2 m/|0| Li(p) (l + |Z fc |f +1 +E[|Z fc |f +1 ]), 
w/iere $j = 4>(Z j+jl ,. . . , Z i+im ) /or a// j > 0. 

Proof. Define H k , r as the M d — > M function mapping x G M d to 

E[^>(/lfe,r+j 1 (x, U fc+ l , . . . , Ufe+y.+jj ),..., h k>r+ j m (x, U fc+ l , . . . , Ufc +r +j m ) )] , 

where, for all ij'GN and (x, ui, . . . , Uj) G M d x M^', 

i 

^ij(x, ui, . . . , Uj) = a(i + j, i)x + ^ a(f + j, i + k)B i+k u k 

fc=i 

l+m 

with a(Z, I) = I and a(Z + ?rt, /) := A k , I > 0, m > 1. 

fc=/+i 
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We thus have, for all k,r > 0, 

(26) E^[^ k+r ]=H k>r (Z k ) 



Using (22), observe that, for all i, j G N and (x,y, ui, . . . , Uj) G M x 
|/iij(x,ui, . . . ,Uj) - hi,j(y, ui, ■ • ■ ,Uj)| < Cp>\x - y|, 



\h itj (x, u u . . . < C U>| + ^- fc £> fc | • 

V fe=i / 

Using these bounds with the Minkowski inequality, the definition of H kjr 
above and the assumptions on (f), we easily obtain, for all (x,y) G M. 2d , 

\H k , r (jc) - H ktT (y)\ < Cl |^| Li(p) (E^'M l x " yl 



l + ^P (r+il)p (|x| p + |y| 



i=i 



m /r+ji > 

+2E E^*"* 5 ^ 

i=i\fe=i y 



|^fc,r(x)| < Ci\(f)\ Li ( p) 

m 

1 + y" /9 (r+ji)(p+l)| x |p+l 



i=l 



m /r+ji \P+1 

+E E/ +ji " tBiu ; + i 
i=i \fe=i / 

where ci is a constant depending only on C, p and p. Observing that 
ELi ^" fc ^ V(l - P) and E™i/^ Q < V(l " P a ) for all j > 1, = j! < 
• • • < 3m and a > 0, we obtain 

(27) |iJ fe>r (x)| <C 1 |0| Li(p) m(l + p^+ 1 )|xr 1 ), 

(28) |ff fc , r (x) - F fe , r (y)| < dm/|0| Li(p) |x - y|(l + ^(|x|f + |y|f )), 

where Ci is a constant depending only on £?*,U* +1 , C, p and p. Equa- 
tion (25) follows from (26) and (27). Observe now that, for any probability 
measure £ of M. d , 

flfc, r (x)- | iT v (y)<(dy) < | |fl- fc)r (x)-H fc , r (y)|C(cfy) 

^Cim/I^Li^) | |x-y|(l + |x| p + |ynC(dy), 
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where we have used (28). The two last bounds of the proposition follow by 
using (26), |x — y| < |x| + |y| and by choosing £ respectively equal to the 
conditional distribution of given (Zo, Ui, . . . , Lb) and to the distribution 
of Z fc . □ 

4. Proof of Theorem 2. We denote by and the space of d x d 
real matrices and the subspace of positive semi-definite symmetric matrices, 
respectively. For all A G MLj, we let X m i n (A), A max (A) and |A| max (^4) denote 
the minimum eigenvalue, the maximum eigenvalue and the spectral radius 
of the matrix A, respectively, that is, A m i n (j4) := min(sp(^4)), A max (j4) := 
max(sp(vl)) and |A| max (^4) := max |sp(^4)|, where sp(^4) denotes the set of 
eigenvalues of A. The proof is derived in five steps and relies on intermediary 
results which will be repeatedly used throughout the paper. 

Step 1. Exponential Stability of the TVAR model. We have the following. 



Lemma 12. Let < po < P an d L be a positive constant. Then there 
exists C > such that, for all A G with \A\ < L and |A| max (^4) < po, and 
for all keN, \A k \ <Cp k . 



PROOF. We apply Theorem VII. 1.10 of [10]. Let 7 = {z G C : \z\ = p}. 
Then for any A such that |A| max (A) < po and for all k £ N, 



Let G := {^4 G MLj : \A\ < L, |A| max (A) < p }. G is a compact set and (z, A) 1— > 
\(I — A/ z)~ l \ is continuous over 7 x G so that it is uniformly bounded. The 
proof follows. □ 

For all t G [0, 1] and for all = (6>i, . . . , 9 d ) : [0, 1] -» R d , let @(t, 0) denote 
the companion matrix defined by 




Hence, putting z = pe iy , y G (— tt, it), we have 




9. 



(29) 



e(t,0) 



1 





1 




















1 
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whose eigenvalues are the reciprocals of the zeros of the autoregressive poly- 
nomial. Using this notation, (2) can be rewritten as 

(30) X k+ ^ n = @(k/n,e)X kjn + <T k+ ^ n £ k +i, n , 0<fc<n-l, 
where <Tk,n = [<r(k/ri) • • • 0] T . 

Proposition 13. Let (3 G (0, 1], L > and < p < r < 1. Then there 
exists a constant M > such that, for all 6 G A^(/3,L) n S(p) and < k < 
k + m <n, 



(31) 



fc+m 

n 

Z=fc+1 



Proof. For notational convenience, we use A and 5 as shorthand nota- 
tion for Aa(P, L) and <S(p). First note that G* = sup 0eAn5 sup tg r 01 ] |0(i, 6)\ 
is finite. For any square matrices A\ , . . . , A r we have 



l[A k = Al + (A r -A 1 )A[- 1 

r 

+ A r (A r -! - Ax)A r 1 ~ 2 + • ■ • + II A ^2 - Ax)A x . 



k=3 



By applying this decomposition to 



k 

(3 n (k,i;0):= [J e(j/n,0), 0<i<k<n, 
j=i+i 

(33) 

fi n (i,i;0)=I, 0<i<n, 
we have, for all < k < k + q < n, 

\p n (k + q,k;9)\<\e((k + l)/n,6) q \ 

+ qQ* q ~ 1 m&x\0{{k + j)/n)-0({k + l)/n)\. 

Let us set p G (p, r). Hence, for all G S(p) and t G [0, 1], we have |A| max (B(t, 
0)) < P- Since 0* < oo, by Lemma 12 we obtain 

sup sup \O q {t,0)\ < Cp q , q G N. 
0eAnS te[o,i] 

Observe that, for all n > 1, 

sup max max |0((A; + j)/n) — + 1)/^)| < L{q/nf. 
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Pick q and then N large enough so that Cp q < r q /2 and qQ* q ~ 1 L(q/N)^ < 
T q /2. The three last displays then give, for all n> N, 

sup max \P n (k + q, k; 0)\ < r q . 
6>GAn<S°< fc < («-<?) 

Write m = sq + t, < t < q. For all n> N,l G {1, . . . , n — m}, and for all 

0G Ans, 

s-l 

|/3 n (/ + m, I - 1; 0)| < G** J] |/3 n (Z + (i + l)q, I + iq; 0)\ 

i=0 

< (l + 9*/r) 9 r m . 

The proof follows. □ 

From Proposition 13 and Proposition 10, we get that under (Al), 

(34) sup sup ||Xfc )n || g) e )0 . < oo. 

(6,cr)eC0<k<n 

Equations (31) and (34) are referred to as uniform exponential stability and 
uniform L q boundedness, respectively. The bound (34) may be extended to 
conditional moments as follows. 

Proposition 14. Assume (Al) with q>l and let p G [l,q]. Let (3 G 
(0,1], L > 0, < p < t < 1 and < a_ < o+. Then there exists a constant 
M such that, for all (6, a) G C(j3, L, p, cr_, <r+) and 1 < k < I < n, 

(35) |E^"[|X,, n |P]| <M(l + r( i - fc )|X M n, 
where T k ^ n = cr(X 0)n , X j>n , l<j<k). 

Proof. Equation (31) is satisfied by Proposition 10. Then under (Al) 
we apply Proposition 11 with 0(x) = |x| p , since <p G Li(M d , l,K;p — 1). Equa- 
tion (35) follows from (25). □ 

Step 2 . Error decomposition. When studying recursive algorithms of the 
form (3), it is convenient to rewrite the original recursion in terms of the 
error defined by <5fc jn := Ok,n(^) — 6k,n, < k <n. Let us denote, for all v > 
and x G R d , 

(36) Li/(x) := - — X . l9 and F„(x) := L„(x)x T . 

1 + z/|x| z 
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The tracking error process {<5fc in ,0 < k <n} obeys the following sequence of 
linear stochastic difference equations. For all < k < n, 



(37) 





— °k,n 


, r(v) , r(w) 
^ °k,n ^ °fc,n> 






°k+l,n 


■= (I- 


■ ^(x M ))4^ 






°k+l,n 


■= (I- 


■^F M (X, in ))5g 




°0,n — u ' 


°k+l,n 


■= (/" 


-/xF^X^g 


+ (0Jfc,n — 0fc+l,n); 


°0,n — u - 



{^fcni takes into account the way the successive estimates of the regression 
coefficients forget the initial error. Making a parallel with classical nonpar a- 
metric function estimation, 5^ plays the role of a bias term [this term 

cancels when the function 1 1— > 6{t) is constant], whereas 5^ is a stochastic 
disturbance. It should be stressed that the "bias term" is nondeterministic 
as soon as 1 1— > 9(t) is not constant. The transient term simply writes, for all 
< k < n, 

(38) 4 u i in = ^(fc,-l;/x)4 u) , 
where, for all fi > and —1 < j < k <n, 

k 

(39) * n (j,j ;/*):= I and ¥ n (fc, j; fi) = JJ (J - ^(X,, n )). 

Let us finally define the following increment processes, for all < k < n, 

dvn := ®k,n ~ Ok+l,n and Cfc^n := / i L At (X fc]n )cr fc+1)n e fc+1))1 . 

According to these definitions, {^fc V ^}i<fc<n and {^^}i<fc<n obey a generic 
sequence of inhomogeneous stochastic recurrence equations of the form 

*&,n = ( J - + e£> = £ i; . o < k < n. 

i=0 

(40) 

In view of (38) and (40), it is clear that the stability of the product of random 
matrices ^ n (k,i;fi) plays an important role in the limiting behavior of the 
estimation error. 

Step 3. Stability of the recursive algorithm. The following stability result 
for the product ^ n (k,j]/i) defined in (39) is an essential step for deriving 
risk bounds for the estimator n . 
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Theorem 15. Assume (Al) with q>4. Let f3 G (0, 1], L > 0, < p < 1, 
and < cr_ < a+ . Then for all p>l there exist constants M, 5 > and fj,o > 
0, such that, for all < j < k < n, fj, G [0,/xo] and (0,<r) € C((3,L, p,a-,a + ), 

(41) ll*n(fc,j;M)IUt>,«T < Af(i - <J/x) fc_j - 

Similar stability results have been obtained in the framework of classical 
recursive estimation algorithms (see, e.g., [13, 20]), but cannot be applied 
directly to our nonstationary and nonpar ametric context. Let us sketch the 
main arguments of the proof in a more general context. Let {Ak{y),k > 
0, v > 0} be an M^~-valued process such that 

(C-l) for all ke N and v € [0,i^], \vA k (v)\ < 1. 

Here Aq{v) , A\(v) , . . . correspond to the matrices F l/ (X J+ i >n ), F^(X J+ 2,n) 5 • ■ • , 
which appear in the product ^> n (k,j;v) for some fixed j, k and n. Taking 
p = 1 in (41) [this can be done without loss of generality under (C-l)], we 
want to prove that E| nl=i(^ — ^^4fc(^))| £ C(l — 5v) 1 for some positive con- 
stants C and 5. 

First observe that it is sufficient to have E :Ffe [|J — iMfc + i(z/)|] < 1 — 8u or, 
more generally, for some fixed integer r and a > 0, 

k+r 
l=k+l 

To obtain this inequality, we expand the product and then use Lemma C.l 
so that 



(42) 



E 



k+r 

n v 

l=k+l 



uA t (u)) 



<l-u\ D 



' k+r \ 

a=k+i / 



where 1Z is some remainder term which will be controlled by using 
Y^d=k+i \Al(v)\ s for some s > 1. It turns out that, in the previous display, 
the conditional expectation of the RHS given may be controlled only on 
a set {(f> k < R±}, where R± > and {4>k} is a positive adapted sequence (in 
the TVAR context, (f)f. corresponds to |Xj. n |). This yields the following two 
conditions: 



(C-2) there exists a\ > such that, for all k G N and v G [0, v{\, 

k+r 

Aminl V) AAv) I >a{L{6 k <R\\ 



E 




(C-3) there exist s > 1 and C\ > such that, for all k G N and v G [0, ui], 

i(<P k <Ri) E ^ k [\Mv)\ s ]<Cv 

l=k+l 
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In turn, (42) may be shown only on the set {cftk < R±}, and it remains to 
check that this happens for sufficiently many k's. For this we use a classical 
Lyapounov condition, namely, 

(C-4) there exist A < 1, B > 1 and an adapted process {V k , k > 0} on [1, oo) 
such that, for all k G N, 

E^ fe [V k+r ] < \V k l{4> k >Ri)+ BV k l{4> h < R{). 

The previous arguments yield the following general result, whose precise 
technical proof is postponed to Appendix A for convenience. 

Theorem 16. Let (0, JF,P, : k G N}) be a filtered space. Let {4> k ,k> 
0} be a nonnegative adapted process and, for any v > 0, let A(u) := {A k (v), 
k > 0} be an adapted Wl^ -valued process. Let r > 1, R± > and v\ > such 
that (C-l)-(C-4) hold. Then, for any p> 1 there exist Cq > 0, 5o > and 
uq > depending only on p, v\, a\, R±, C\, s, B and A such that, for all v G 
[0, i/o] and n > 1, 

Having this general result in hand, we now prove Theorem 15. 

Proof of Theorem 15. To verify (C-2), which is referred to as the 
persistence of excitation property in the control theory literature, we need 
some intermediary results which hold under the assumptions of Theorem 15. 

Lemma 17. There exists C > such that, for all d< j <n and (0,o~) G 

C(f3,L,p,<T-,<T + ), 

(43) A min (E , CT [F,(X J - „)]) > C, v G [0, 1], 
and, for all < k < j — d, 

(44) A min (Ej-[F (X J - n )])>C, 

(45) X min (M^[F u (X jtn )]) > ° v G [0, 1]. 

Proof. Let x G M d , |x| = 1, d<j<n, and (9, a) G C := C((3,L,p,a-, 
a + ). Write 

d 

^-j,n = (3n(j,j - d; 0)Xj„ d rl + y^^P n (j,j — d + i] 6)Oj_ d+itn ej_d+i,n, 

i=i 



i=l 



<C e 



-Sown 



V . 
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where n is defined by (33). We have E J" d '" [(x T X iin ) 2 ] > a 2 x T H jt n ,d( e )^ 
where Hj^ n ^{9) := Cj, n ^{0)Cj nd {6) is the controllability Gramian (see [17]) 

C,>, d (0) := [Pn(j,j - d + 1; 0)G • • • (3 n (j,j; 0)G\, where G := [1 . . . 0] T . 

One easily shows that, for i = 1, . . . , d, (3 n (j,j + 1 — i; ff)G has a unit ith coor- 
dinate and zero coordinates below. Hence, det(Hj in> d(9)) = det{Cj n ^{0)) = 
1. In addition, from exponential stability we have, for all (0, a) G C, \Cj t n t d(0)\ < 
M for some positive M not depending on (j, n). Hence, for all d < j < n and 
(«,*)€ c, 

Amax(,-ri : ,>,dV t 'Jj 

It follows that, for all < A; < j - d, x G M d such that |x| = 1 and (0, a) G C, 

(46) x t e£T [Fo(X jire )]x = E [e£— [(x t X,- n ) 2 ]] > a 2 _M^ d ~ l \ 

showing (44) for any C > cr 2 A^"^" 1 ^- Equation (43) also follows for v = 0. 
For all < k < j — d and for all v G [0, 1] , write 

E& n [(x T X,>) 2 ] = E%» [ (1 + '^l )1/2 (1x^^1(1 + HX^l 2 ) 1 / 2 } . 
The Cauchy-Schwarz inequality shows that 



x T E^ n [F-(Xi,n)]x>^ 



(^"[(^X^n) 2 ]) 2 



Egr[(xrx iiB )2(l + i/|X iiB |a)] 
Since |x T X Jjn | < |Xj )n | for |x| = 1, by applying (46) we get, for all v G [0, 1], 
r kn (a 2 _M-( d ~V) 2 

A m inQE '"[FvpLj n )]) > —= ~ . 

^"[ix^h+e^-iix^i*] 

The proof of (45) then follows from (35). The proof of (43) is along the same 
lines. □ 

Lemma 18. Let 4> G Li(M d , 1). There exists M > such that, for all 
1 < i < k < n, u>0 and (9, a) G C((3,L,p,a-,a+), 

k 2 n 



J2 WX J> )-E^[0(X J> )]) 

J"=i+1 



<M(fc-i)|^Li(i)(l + |X i:re | 



Proof. For j G {t + 1, . . . , k}, denote Aj = </>(X.j, n ) - E^jT [<K x j>)] • For 
all * <j < I < n, e£>[A,] = E£>[0(X,, n )] - E^[</>(X,, n )j. The model is 
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uniformly exponentially and L q stable [see Proposition 13 and (34)]. From 
Proposition 11 there exist r G (p, 1) and Ci, C2 > such that, for all < i < 
j<l<n, (6,a)£C{f3,L,p,a-,a + ), 

|Efc n [A ; ]| < dr^'|0| Li(1) (l + |X Jin | 2 + |X iin | 2 ), 

|E£' l [A 2 ]|<C 2 |0| 2 i(1) (l + |X 4 , n | 4 ). 

From Proposition 14 we get that there exists C3 > such that, for all < 
i<j<l<n, (0,a)£C(/3,L,p,a-,a+), 



< (Eg; [AjD^CBgr [(Ej> [A,]) 2 ]) 1 / 2 

<C 3 |0|L(l)T'- i (l + lXi.nl 4 ), 



and the result follows. □ 



Lemma 19. For all Ri,a\ > 0, there exists ro > 1 suc/i i/iai, /or all 
(0,a) G C(f3,L,p,a-,a+), r>rQ, n>r, k = 0, . . . ,n — r and v G (0, 1], 



(47) 



E 



0,0- 



A, 



/ k+r \ 
\l=k+l ) 



> aiI(|X fc , n | < 



where F u is defined in (36). /n addition, there exist constants 5 > and 
/^o > 0, suc/j i/iai, for all (6, a) G C(f3, L, p, cr_, + ), d< k <n and v G [0, z/q], 



(48) 



PROOF. For any symmetric matrix A, we have |A m i n (A)| < |A| max (^4) = 
\A\ (recall that | • | denotes the operator norm) and X m i n (A) = inf| x | =1 x T Ax. 
From the last assertion, it follows that, for any symmetric matrix B having 
the same size as A, A m i n (A + B) > A m i n (A) + A m i n (-B). Therefore, 

Amin(A) > X min (B) + Amin(^4 - B) > A min (£) - \A- B\. 

Applying these elementary facts, we get, for all0</c<fc + r<ra, 



/ k+r \ 

Aminl ^"(Xj'.n) J 

\i=A;+l / 



\j=k+l 
k+r 



> E A^E^F^X^)]) 

j=k+i 



k+r 



E (F,(X Jin )-E^[F,(X,, n )]) 
j=k+i 



From its definition in (36), F^(x) G for all x G M. d and v > 0, and 



(49) 



sup|F„| Li(1) < 00. 

v>0 
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Applying Lemmas 17 and 18, we obtain that there exist C, M > such that, 
for all (9, a) eC and 0<k<k + r<n, 



E 



A, 



( k+r 
\j=k+l 



> 



C(r - d) 
1 + |Xfc J- 



MVF(1 + |X fc) „ 



4n1/2 



Now pick two positive numbers -Ri and cti. If |Xj. n | > Ri, equation (47) 
is clearly satisfied. If |Xfc n | < R\, the last equation implies that, for all 
< k < k + r <n, 



A, 



f k+r 
\j=k+l 



> 



C(r - d) 
l + Ri 



M^l + Rf) 1 / 2 . 



We may thus find ro such that the RHS of this inequality is greater than or 
equal to a% for all r > ro- This concludes the proof of (47). 

From the uniform L 2 -boundedness and (49), we get that there exists M 
such that, for all (0,a) G C, v € [0, 1], d < k < n, \E 0>a [F v (X kjn )]\ < M. Thus, 
using Lemma C.l, for all v G [0, 1/M], d< k <n, and (6, a) G C, 

\I-uE 9>cr [F v (K ktn )}\ = 1 - v\ min (E e ,AFu(Xk,n)}) 
and the proof of (48) follows from Lemma 17. □ 



We now turn back to the proof of Theorem 15 by applying Theorem 16 
to the sequence 

{{Ai = F v (X j+1+lin ), F j+1+ i iU ), I = 0, . . . ,k - j - 1}. 

It thus remains to show that conditions (C-l)-(C-4) of Theorem 16 hold with 
constants r,Ri,ui,a±,Ci,X,B and s, which neither depend on (0,a) GC 
nor on j, k, n. Set a\ = 1 and, for I = 0, . . . , k — j — 1, V\ := 1 + |Xj + j >n | and 
4>l '■= |Xj+z, n |- 

Condition (C-l). For all v > and x G R d , ^^^(x)! =z/|x|/(l + z/|x|) < 1, 
which yields (C-l). 

Condition (C-2). From Lemma 19, we may choose ro depending only 
on Ri and ot\ such that (C-2) holds for all r > ro- 

Condition (C-3). From Lemma C.2, sup l/>0 1 1 | <?/ ' 2 1 Li(<j i) < oo. From 

Proposition 14, there exists M > such that, for all v > 0, < i < i + I < n 
and (6, a) G C, 

E^ n [\F u (Xi +l>n )\^ 2 ] < M(l + r"|X iin |«). 
Hence, (C-3) is obtained with s = q/2 and C\ = Mr(l + Rf). 
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Condition (C-4). Let r £ (p, 1). From Proposition 14, there exists M such 
that, for all < i < i + r < n and (0,<r) £ C, E^ a n [\X i+r , n \] < M(l + r r |X; in |); 
thus, for any Ri > 0, 

E^" [1 + |X i+r , n |] < Mr r |X iin | + M + 1 

< ^Mr r + ^±il(|X 4 ,„| >i?!)) 

X (l + |X i , n |) + (M + l)I(|X i , n |<lJi). 

Choose r > r and i?i > so that Mr r + (M + 1)/J?i < 1. Condition (C-4) 
is then satisfied with A := Mr r + (M + l)/R\ < 1 and B = 1 + M. 

Finally, we obtain, for some positive constants r, Cq,5o and fiQ, for all 
v £ (0, //o], (6, c) £ C and < j < k < n such that n — j >r, 

ll*n(fc,j»||^ i(T = E[Ej; +1 -[* n (fc,j»]] 

< Coe 5oim (i + ||Xj + i )n || Pi )<T ). 

The uniform boundedness (34) then yields (41) when n — j >r. The restric- 
tion n — j>r above is needed because (C-4), (C-2) and (C-3) are well defined 
only for n — j <r. Now recall that (C-l) implies \^/ n (k, j; u)\ < 1. The result 
for n — j <r (implying k — j <r) follows by taking M > (1 — 5z^i)~ r in (41). 
□ 

(Step 4. Error bounds. Similarly to (49), one easily shows that 

(50) sup | | Lift)) < oo, 

where L is defined in (36). The L 9 -boundedness of {Xj. n , 1 < k < n} then 
gives 

( 51 ) F *q/2 : = SU P SU P SU P \\ F u0^k,n)\\q/2,e,a <00, 

(0,cr)eCf>OO<fc<n 

(52) L* := sup sup sup ||L„(Xfc ifl )|| g) )(T < oo. 

(0,cr)eC v>0 0<k<n 

From now on, for convenience, we let the same M, 5 and hq denote positive 
constants depending neither on the time indices i, j, k,n,..., the step-size \x 
nor on the parameter (0,o~). 

Applying (38) and (41), for all u > 1, there exist M > and /j,q > such 
that, for all pi £ (0, /j,q\ , (0, a) £ C and 1 < k < n, 

(53) \\^Xe,a<M(l-S^ k \e(0)\. 
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Define 



k-l 

3W(fc,fc):=0 and E$(k, j) := £ C>> 0<j<A:<n. 

For all 1 < j < k <n, we have 

tf n (& - l,j» - * n (A; - 1, j - = fi¥ n (fc - l,j;^)F /4 (X i _ 1 , n ). 
By integration by parts, for all 1 < < n and /x G (0,/xo], (40) reads 

^ = $ n (fc-l,0 ;/ i)HW(fc,0) 

fc-i 



(54) 



By applying the Holder inequality and using (41) and (51), we get that, for 
any u G (l,g/2), there exists M > such that, for all /i G [0,/io], G C 

and < j < k < n, 

(55) ||* n (fc,j;/i)F /4 (X i _i )n )|| Ui( , )<r < M(l - 5^ +1 . 

We consider now the two terms 5^ and 5^ separately. We have, for all 
< j < k < n, 



(56) \Zt\k,j)\ 



k-l 

i=j 



(w) 
i.n 



\Oj,n-Ok,n\<\O\A,f>n-0(k-jf. 



Inserting (56), (41) and (55) into (54), we thus get that there exist M > 0, 
5 > and /Uq > such that, for all (0, a) G C, I < k <n and /i G (0, /jlq], 



k-l 



IICIU. < Ml (1 — ^(fc/")" + ^(1- <^'((fc - j)/n) 

V j=i / 

By Lemma C.3, picking u>p, we obtain, for all /i G (0,/io] and (0,cr) G C, 
(57) IIOp^^M/^K !< fc < 



n. 



We finally bound 5^ v \ Note that, for each n > 1, {o"i jri L M (Xj jn )ej_|_i in , 
i = 1, . . . , n — 1} is an J^+i^-adapted martingale increment sequence. The 
Burkholder inequality (see [14], Theorem 2.12) and (52) give, for all (0,a) G 
C and /i > 0, 



(58) 



< M(k - j + 1) 1/2 , l<j<k<n. 
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By Lemma C.3, using (54), (58) and (55) with u such that l/p = l/u + l/q, 
we get, for all /i G (0, no] and (0, a) G C, 

(59) ¥ { kl\\ P ,9,a<M^ l<k<n. 

Equation (13) easily follows from (37), (53), (57) and (59) and Theorem 2 
is obtained. 

5. Proof of Theorem 4. By writing 5 n := u T S n , (14) simply means that, 
for all real-valued estimators <5 n (Xo )n , Xi^ n , . . . , X Utn ) and u = (ui, . . . , u^) G 
R d such that |u| = 1, 

sup E 0)CT [|4 - u T 0(t)\ 2 } > an-W^y 

(<V)eC 

Denote by \J3\ the largest integer strictly smaller than (3. Let 4> : M — > R be a 
C°° symmetric function decreasing on M + such that c/>(0) = 1, <j){u) = for 
all \u\ > 1 and 

(60) SUp j r™, < 1. 

Let A : R — > be a C 1 p.d.f. with respect to the Lebesgue measure vanishing 
outside the segment [— 1,+1] and such that 

(61) £(H)V),fc<=c, 

where A is the derivative of the density A. Let (v n ) and (w n ) be two nonde- 
creasing sequences of positive numbers to be specified later such that 

(62) lim (v~ 1 + uC 1 + n~ l w n ) = and supt> < L. 

n ^°° n>0 

Let t G [0, 1] and u G JR d such that |u| = 1. Define : [0, 1] -> M d , s i-> 
4>n,t(s) '■= <p((s — t)w n )u and let cr : [0, 1] — > M + , s i— > <r(s) = <r+. For n > 1, 
define 

(63) X fc+ i >n = r](j)((k/n - t)w n )u T X k:n + (cr + )e fe+ i in , < fc < n - 1. 
From (A2), it follows that, for all < k < n — 1, 

X i-^Pfc+l,n((a; - V4>n,t( k / n ) uT y)/ (J +)/ (J + 

is the conditional density of X^+i n given X& n = y and parameter r/. Since 
the distribution of Xo, n does not depend on 77, and using the conditional 
densities above to compute the joint density of {X.o ;n ,Xx >n , . . . ,X n>n }, the 
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Fisher information associated with the one-dimensional parametric model 
defined by (63) is 



T n {j]) = cr + 2 E^ n a 



0n,t(^/n)u T X fc _i in ^^(e fc 



vfc = l 



Pk,n 



Now, under (A2), the summand in this equation is a martingale increments 
sequence whose variances are bounded by 2 t (fc/ra)E^ n ti<J [(u T Xfc_ ljn ) 2 ]Z e . 
Since |u| = 1, (u r X) 2 < |X| 2 , and we finally obtain 

n 

(64) l n ( V ) < afZ e 2 it (/c/n)E^ !t , CT [|X fc _ lin | 2 ]. 

k=l 

From (60) and (62), for all 77 G [-tJ^ 1 ,^ 1 ], 

. w;F J |0(^J)((^-tK)-^^)(( S -tK)| ^ «;g . _ 

#n,fA,)3< SUp — — -r-r, < <L, 

and \r]4> n! t(0)\ < v" 1 . Hence, for large enough n, rjcftnj G Ad((3,L) for all 77 G 
[— u^" 1 ]. By construction, for s £ [0,1] the autoregressive polynomial of 
J?0n,* is given by 1 - T}<j> n> t{s) Yh=i u i z * '■ Since lim^oo v n = 00, for any p, < 
p < 1, there exists iV, such that, for all n > N, i](ft n j G 5(p), 77 G [— v~ ,v~ l ] } 
and, thus, (77^,(7) G C. Using (34) for bounding tjCT [|X fc _i jn | 2 ] in (64), 
it follows that there exists M depending only on p,[3,cr + and L such that, 
for all sufficiently large n and for all 77 G [— v~ x ,v~ ], 

2n(??) < MT e X! 4> 2 {kw n /n - tw n ). 

kGZ 

Using the fact that (ft is C 1 and compactly supported, we have 



lim sup h 



J2<ft\kh-x)- [ cft 2 (t) 



dt 



0. 



Equation (62) shows that, for large enough n and for all r\ G [— v n ,v n l ], we 
have 



(65) 



I n (ri) < Ml e nw- 1 J <ft\t) dt(l + o(l)). 



We get that, for all real valued estimators 5 n := j n (Xo, n ,-X"i )n , . . . ,X njfl ), as 
n — > 00, 

_ 7j n A(w n ,?7)E r;(/)n t ,a[(5 n - t]) 2 ] dq 

-v n L 



> 



sup T n {rj)+l n (\) 
>(0(nw- l +vl)r\ 
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where the first inequality is the Bayesian lower bound of the minimax risk 
[recall that, for n sufficiently large, (r]4>n,t, cr) £ C for all |r/| < v~ l ], the second 
inequality is the so-called van Trees inequality (see [11]) with 2^(77) denoting 
the Fisher information of the translation model associated with the p.d.f. 
v n ^{v n -) and the last inequality is implied by (61) and (65). The proof is 
concluded by choosing v n = w@ and w n = n l /^ l+2 ^ . 

Remark 7. Theorem 4 easily extends to cases where the distribution 
of {X-o tn ,n > 1} depends on in a not too pathological way. Assume, for 
instance, that Xo in follows the distribution of a stationary AR process with 
parameter (0(0), a (0)) and with a given white noise (see, e.g., [6]). In this 
case, the lower bound (14) holds for t > and n sufficiently large without fur- 
ther assumptions. This clearly follows from the proof: since the distribution 
of Xo jn depends only on (0(0), a (0)) and since, for t > and n sufficiently 
large (r](j) nt t(0),a(0)) = (0,0+) does not depend on 77, the computation of 
I n (77) applies and the proof proceeds similarly. 

6. Perturbation expansion of the error. In this section we first derive 
several approximation results for the error terms 5^ and $M defined by (40) 
in Section 6.1. Computational estimates are then obtained in Section 6.3 
and the proofs of Theorems 6 and 7 are finally given in Section 6.4 and 
Section 6.5, respectively. 

6.1. General approximation results. Observe that Theorem 2 only pro- 
vides a bound of the risk. The approach developed in this section relies 
upon a perturbation technique (see [1]). Decompose the LHS of (40) as 
C = 4'f + with J#» = 0, <°) = and 

4+Z = (I- ^ , a [F,(X k , n )])4'f + £jM 

Ek+i, n = i 1 - /^(X*,n))ff& 0) + M(E , CT [F M (X fc , n )] - F M (X M )) j£; 0) . 
The inhomogeneous first-order difference equation satisfied by jj?^ yields 

k 

(66) 4+i!n = E Mk, i; M> 0, , < k < n, 

i=0 

where, for all fi > 0, < % < k < n, and (0, a), 

k 

ip n (i,i;ii,0,a) := I and Tp n (k,i; n,0,a) := JJ (I - fjE0 ta [F^(X j>n )]). 

j=i+l 

For ease of notation we write F fci „ := F M (X fc) „) ) F fcjn := F fc>n (/j)-E 0)CT [F fc|n (/j)], 
E instead of Eg a , ^ n (k, i) instead of ^ n (k, i', /-*), ipn(k, i) instead of ip n (k, i; \x, 
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6, a) and so on. This decomposition of the error process {#1*1, 1 < k < n} 
can be extended to further approximation order s > as follows: 

( 67 ) 4'n = J fc'n 0) + 4'n ) 1" 4"n ) + ^ ' 

where 

jftt = (' - ^[f m ]) j£f + e« , jg,o) = 05 

4rl = (i- ^[F fc , n ])4x° + ^UT 11 - = o. 

< Z < r, 



Z = 0,...,s. 

The processes ^ depend linearly on Q, ^ and polynomially in the error 
Fjfc,n- We now show that j( w >°) and j( v '°\ respectively, defined by setting 
= £( w ) and = £^ i n (66), are the main terms in the error terms o~( w ) 
and 5^ defined by (40). 

Proposition 20. Assume (Al) mi/i g > 4 and Zei p G [l,g/4). Lei /3 G 
(0,1], L > 0, < p < 1, and < a_ < cr+. TTien i/iere exist constants M and 
po > 0, such that, for all (0, a) EC((3,L, p,a + ,a-), p G (0, po] and 1< k <n, 

(68) |4 w n 0) |<M(np)^, 

(69) W^l-4T\U^ M ^(n^. 

Proof. From (48) in Lemma 19 (which holds under the assumptions 
of Theorem 15), there exist 5 > 0, po > and M > such that, for all 
< i < k < n, p G [0, p ] and (0, a) G C, 

(70) |V>„(A;,i;//,e,(7)| <M(l-5p) k ^. 

Note that ip n {k,i — 1) — 'ifj n (k,i) = —pil) n (k,i)M\F^ n ]. As in (54), write, for 
all 1 <k <n, 

(71) j£ 0) =^ n (fc-l,0)E4 w ^^ 
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Using (56), (51) and Lemma C.3 shows (68). By (67) we have 6™ - J (w ' 0) = 
H (w,o) = j(w,i) + fl -(w,l) j where, for all 1 < k < n, 

(72) ^^^Mk-hjWj^, 

3=0 

(73) < n 1} = /*£*»(* " Lj^n^- 

3=0 

Set <^-(x) = ^ n (fc- 1, j')F /J (x) J^' 0) , j = 0, . . . , jfc-1. Note that, from (49), (70) 
and (68), for all fj, G (0, /x ], < j < k < n and (0, a) G C, 

l^lLi(i) < \Mk ~ 1, j)ll4n 0) H F MlLi(i) < M{1 - S^) k -i(nn)-P. 

By applying Proposition B.2 componentwise, we get, for all \x G (0,//o], 1 < 
k < n and (9, a) G C, 

/jfc-1 \ 1/2 

<m(h _/j ECi-ty)* -1 -'' • 

g/2 Vj=0 / 



E^(fc-l,j)Fi, B 4j 0) 



Hence, for all \i G (0, /zo] > 1 < & < n and (0, o") G C, 
(74) HJ^II^^M^W^. 

Let « be such that 2/q + 2/q + l/u = 1/p. Thus, by Theorem 15 and (51), 
for all fx G (0, /xo], 1 < k < n and (9, a) G C, 

fe— l 

< ||*»(* - i,j)IW|F ilB || B/2 || j$\ /2 

3=1 

(75) 

fc-i 

< M^ 2 {nn)-P £(1 " ^) fc_i < M^Ji{^n)- p . 

3=1 D 

PROPOSITION 21. Assume (Al) mi/i g > 7 and let p e [l,2g/ll). Lei 
/5 G (0, 1], L > 0, < p < 1, and < er_ < <r + . Then there exist constants M 
and fj-o such that, for all (9, a) G C({3, L, p, cr_, a + ), fi G (0, po] and 1 < k < n, 

(76) UT^KA^M^ 

(77) llC-4? } ^^)IU CT <^- 

Proof. The Burkholder inequality (see [14], Theorem 2.12) shows that, 
for all 1 < k < n, \i G (0, po] and (9, a) G C, 

/fc-i \ i/ 2 /fc-i \ I/ 2 

\\4 V f\\ q < M»a + L*e*\^2jMk ~ 1, j)| 2 J < ^(£(1 - ^) 2 J 
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and (76) follows from (52) and (70). We now bound 
fc-l 

Jk,n = M £ ~ IjW^f, 1 < k < n. 
3=0 

Let us pick 2 < k < n. By plugging (66) with £^ = we obtain 

(78) 4 V ' } =» 2 £ 0i(X j ,n)7ij(X i ,„)<7 

0<«<j<fc-l 

where, for all < z < j < A; — 1, 

(79) <j)j(x) :=ij) n {k- l,j)F (U (x) and 7 i,j(x) := ^(j - l,i)L M (x). 

From (49) and (50) we have, for all < i < j < k, fa G Li(M d , l,R d x R d ; 1) 
and 7y G Li(R d , l,R d ;0) and, furthermore, from (70), 

|^-| Li(1) < M(l - 8fi) k -i and | 7i)i | Li(0 ) < M(l - 5^~\ 

where, as usual, M and 5 are positive constants, not depending on indices 
k, n, on fx G [0, jlio] or on (6, a) G C. The following uniform bounds follow: 

t-i 

sup \fa\u{\) < M, l0jlLi(i) < 



o<i<* j=i 



1/2 

1/2 



sup |7i,ilLi(o) <Af, sup Nj'Ilko) ) 

o<*<j<t o<J<*Vo<i<j 7 

By applying Proposition B.3 componentwise, we obtain ||^„ H29/7 — -^A* 
uniformly over 1 < A; < n, fj, G [0, /Uo] and (0, a) G C. As in (75), let u > 1 be 
such that u -1 + 2/g + 7/2(7 = 1/p. Then, for all 1 < k < n, {i G [0,//o] and 
(0,ct)gC, 

fc— l 



and thus, H-ff^ ||p < M/i 2 F* /2 E*=i(l ~ <^) fc_i , which yields (77). □ 



6.2. Proof of Proposition 5. We may write, for all 1 < A; < n, 

k 

X fc , n = n (k, 0; 6>)X , n + ]T Pn(k,j; 0)<t 

i=i 

where a and f3 n are defined right after (30) and in (33), respectively. Thus, 
E 0)CT [X fc , n Xj n ] = /3 n (A;,0)E[Xo, n X^ n ]/3 ra (A:,0) T 
fc-l 

+ 2^ Pn(k, k - l)(Tk-l,n(/3n(k, k - l)<T k -l,n) ■ 
1=0 
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Let 6 € S{p) with p > 1, t € [0, 1] and let {Z k , k G Z} denote the sta- 
tionary AR(0(i), cr(t)) with i.i.d. centered unit variance innovations denoted 
by {efcjfcez- Recall that T,(t,6,a) denotes the dx d covariance matrix of 
{Z k ,k £ Z}. Then, using classical results on AR models (see [4]), we have 
[Z k ■ ■ ■ Z k _ d ] T = E/> ® l (t, 0)a(t)e k _i, where a(t) := [a(t) • • • 0] T , 6 is de- 
fined by (29) and the convergence holds in the L 2 sense. It follows that 

oo 

E(t, 6, a) = J2 &\t, e)a(t)(@ l (t, 0)cr(t)) T . 

1=0 

Denote &k,n := @(k/n, 6) and T, k)n := T,(k/n, 6, a). We obtain 
= /3 n (A:,0)(E[Xo, n X^ n ] - E 0l „)A»(fc, 0) T 

(80) 

+ (@n( k > k ~ ^k-lAMk, k - l)(Tk-l,n) T ~ ©L,n cr fc,n( i,n cr fc,n) T ) 
1=0 

oo 

+ ^(l3n(k,0)e^a 0tn (l3 n (k,0)e l -J:a 0tn ) T - Q l k , n a Kn {Q l k ^a Kn ) T ). 

l=k 

Note that, for any matrices A\, . . . , A r and B%, ■ ■ ■ ,B r with compatible sizes, 

(si) fiA-fiB^H'nA^^-B^t n B k ). 

i=l i=l j=l\k=l / \k=j+l / 

From uniform exponential stability (see Proposition 13 and its proof), there 
exists M such that, for all 1 < I < k < n and (9, a) 6 C*, \/3 n (k, k - 1)\ < Mr 1 
and \Q l kn \ < Mt 1 and, thus, 

l-i 

\(3 n (k, k-l)- &l n \ < Mt 1 \Qk- 3 ,n ~ &k,n\ < Mn-^r l f +l . 

3=0 

Similarly, there exists M such that, for all 1 < I < k < n and (0,a) 6 C*, 

\r\l—k fV — ^1 

l U 0,n u k,n I 

<Mrr f, <T i - k (l-k)hP, 
\j3 n (k, k - l)<T k -l,n(l3n(k, k - /)<T fc _; >n ) T - @[ >n <T k,n(Qk, n a ' k,n) T \ 

1^,0)0^ a Q ,n(Pn(k,0)@^a Q , n ) T " ® l k,n<7kA®ln<7k,n) T \ 

< Mn-^T 2l lk p . 
The result follows by inserting these bounds in (80). □ 
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6.3. Further approximation results. To derive tractable asymptotic risk 
estimates, we need to derive approximate expressions for jj^lf^ and jl„ • 
We first derive approximations for E[F^ n ], 1 < k < n, and related quantities. 

Lemma 22. Assume (Al) with g > 4. Let j3 G (0, 1], L > 0, < p < r < 
1, and < a_ < er + . T/ien i/iere exist positive constants 5,vq and M such 
that, for all < k < I < n, v G [0, uq] and (9, a) G C*(/3, L, p, cr_, <r+), 

(82) |(7-^(//n,0,<j))"|<(l-H". 

(83) |S(Z/n, 0, a) - E g>a [F u (K ktn )}\ < M{r k + n~ /3 (/ — k+ if + u), 

(84) |£(Z/n, 0, a) - E^fL^X*. J]| < M(r fc + n^(Z - A; + 1)^ + v). 
PROOF. By continuity of (6,z,t) \- > \9(z;t)\ [see (6)] and since 

jtf GR d :|tf| <L,1- 5^0***^0 far all \z\ < p _1 j 

is a compact set, there exist 5 > and M > 0, such that for all (9, a) G C*, 

\z]=iti[o,i] \0(z;t)\< 

(85) 

|z|=ite[o,i] 

Equation (82) then follows from Lemma C.l. Similarly, there exists M < oo, 
such that for all (9, a) G C* and all < s < t < 1, 

(86) |S(t;»,(7)-E(a;©,(7)| <M(t-s)^. 

By Proposition 5 we get, for all 1 < k < I < n and (9, a) G C*, 
|£(Z/n;0,a)-E[X fe , n x£ n ]| 

(87) < |£(Z/n; 0,<r) - £(fc/n; 0, a)| + |£(A>;0,ct) -E[X fc , n X£j| 

^M^ + n-^Z-fc + l)"). 

This is (83) and (84) with ^ = 0. One easily shows that, for all x£K d and 
v > 0, |F„(x) — xx T | < z^|x| 4 and |L„L„ (x)| < 2^|x| 4 . Since q > 4, we deduce 
(83) and (84) for v > from (87) and uniform L 4 boundedness. □ 

Let p G (0, 1) and let 9 G <S(p) and a: [0, 1] -» M+. Define the following 
sequence of recurrence equations applying to some increment process n , 
0<k<n}: 

J$i, n (fi, °) ■= E( 7 - ^((k + l)/n, 0, a))*-^J, < k < n. 

3=0 



5 < mf inf i ' < A min (£(i;0,a)) 



< A max (£(i;0,a)) < sup sup <M. 
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We now show that J (w ' 0) and j( v '°) may be approximated by J^' and 
JW, respectively, defined by setting £<» = and = £ {v) in (88) and 
then we compute asymptotic equivalents of and of jW's variance, re- 
spectively. 

Proposition 23. Assume (Al) mi g > 4. Lei /3 € (0, 1], L > 0, p < 1, 
and < o~_ < o~+ . Then there exist constants M > and po > suc/t t/iat, 
/or aZZ (0, a) G C*(/3, L, p, o\_, cr+), /i G (0, /Xo] and 1 < k < n, 

(89) \4 w f(e,a)-J^(e,a)\<M(pn)-^(pn)^+p), 

(90) || j£>,o-) - J^(9,a)\\ q>0><7 < M^((pn)^ + /*)■ 

Proof. Let A n (k,j;p,0,a) := ip n (k,j;p,,0,a) - (I - pX(k/n,0, 
f or gji o < j < k < n, p > and (6, a). The definitions of j(*>°) 
and j(') yield 

(91) J& 0) " -55 = J2 A n(k - l<k<n. 

j=0 

From (81) we get, for all < j < k < n, p > and (6, a) G C*, 
fc-l 

A n (fc - 1, j) = A* E ^»(* - l,i + l)(E(Fi, n ] - E(fc/n))(/ - p^k/n))' 1 ' 3 ' 1 . 
i=j 

Using (70), (82) and (83), there exist 5 > 0, /i > and M > 0, such that, 
for all 1 < j < k < n, p G (0, p, ] and (0, cr) G C*, 

(92) \A n (k - 1, j)| < M/i(l - ^-i" V + n~"(fc " + " i))- 
We further write, for all 1 < j < k < n, 

A n (k-l,j)-A n (k-l,j-l) 

= fiA n (k-l,j)E[F^ 1>n ) 

+ p,(I - ^(k/n)f- j - l (nY^ lyn ] - E(k/n)). 

Applying (92), (82) and (83) and observing that is finite, we get that there 
exist 5 > 0, no > 0, r G (p, 1) and M > 0, such that, for all 1 < j < k < n, 
p G (0,po\ and (6, a) G C*, 

\A n (k-l,j)-A n (k-l,j-l)\ 

(93) < Mp(l - 5p) k ^\n^{k-jf{p{k-j) + 1) 

+ p{p{k-j) + l) + Ti). 
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By integrating (91) by parts, for all 1 < k < n, — J^n reads 

k-l 

A n (k - l,0)4")(fc,0) + £(A n (fc ~ U) - A n(k ~ l,j ~ l))E^(k,j). 

3=1 

Using (56) and (58) to bound and ||Hi v ^|| g respectively, together with 

(92), (93) and Lemma C.3, there exist 5 > 0, fiQ > and M such that, for 
all 1 < A; < n, n £ (0, // ] and (0, a) G C, 



+ 1V 3 



k-l 



\\4T - 4% ^ M v^((^) _/3 + m) + m 2 E(! - ^)^ fc_J ' vT+t • 

V j=0 J 

Using Lemma C.3, we have, for any a > 0, 

j2(i-5^T k -^j + ir 

j=0 

<(i-Tr i sup(i-5fiyu+ir 



<C(l-r)-\6^)- a . 
We thus obtain (89) and (90). □ 

Proposition 24. Let (3 £ (0, 1], L > 0, p < 1, and < cr_ < a + and 
let (0,a) £ C*(P,L,p,a_,a + ). Let t £ (0,1] and assume that there exists 
t> p £ R d , V > and j3' > (5 such that (18) holds for all u £ [0,t]. Then 
for all p £ (0,po] and n > 1, 



where M and po are positive constants depending only on (3,L, p,a-,o~ + ,L' 
and P' . 

Proof. Let us write 

f( w ) T 

J [tn],n Jn 
[tn]-l 

= £ (i-mECM/")) 1 *" 1 " 1 "' 

j=0 

x [$? - n-P({[tn) - jf - ([in] - 1 - jf)6 tjP ] 
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where, within this proof section, we denote 
[tn]-i 

J n := n-P £ (I - M £([HA0) ^^'((N - jf - ([tn] - 1 - jf)O t , p . 
j=0 

Using (18), the partial sums of the terms within brackets in the next-to-last 
equation satisfy, for all < j < [tn] — 1, 

[tn]-l 

E KS? - n- p (([tn] - if - ([tn] - 1 - if)O t> p] 

i=j 

= \6(j/n) - 9([tn]/n) + n^([tn] - jfe t ^[ 

<L'n~P'([tn] -jf. 

Integration by parts with this bound and (82), and then Lemma C.3, give 
that there exists a constant M such that, for all u G (0,/io] and n > 1, 

\j}:Xn-Jn\<M(^'. 

Now, from (82) and using Lemma C.3, we have, for all p G (0,/io] an d n > 1, 

\J n -n- p Sp(I-^([tn]/n))e t ,p\ 

< Mn" /3 (1 - 5u) [tn] E(l - ^)'^ _1 
i>i 

^M^V-dp^^pn)-! 3 , 

where 5/j(A) := EiZo^((i + 1)" - i"). Using the fact that A Sp(A) is a 
power series with unit radius of convergence, (82) and (86), from the mean 
value theorem and Lemma C.3, we have, for all p G (0,po] and all n > 1, 

(5/3(1 - /i£([tn]/n)) - S P (I - pZ(t))\ < M/m^^l - <5/i)^ 

i>l 

< M(pn)~ p . 

Collecting the last three inequalities together with Lemma C.3, we obtain the 
result. □ 



Proposition 25. Assume (Al) with q>4. Let j3 G (0, 1], L > 0, p < 1, 
and < ct_ < <7 + . T/ien £/iere exist constants M > and /io > swc/i i/jai, 
/or all (6, a) G C*(/3, L, p, cr_, cr + ), /i G (0, /io] and 1 < k < n, 

a 2 (k/n) 



-I 



<Mp(p+(pn)~ p + (l-5p) k ). 
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Proof. Since {^ v \ n ,j > 0} is a martingale increment sequence, for any 
1 < k < n E[jg jgf] reads 

L* 2 EV " MS(A ; /n)) fe - 1 -V| in E[L^(X i , re )Lj(X, > )](7 - ^(k/n)) k ^ 

3=0 

= f J,<T k,n((*k,n ~ Gk,n) + Rk,ni 
where, for all 1 < k < re, 
fc-i 

G\ n : =/ u E (/-^S(A:/n)) fc - 1 - J S(A : /n)(/- / uS(A : /n))^ 1 ^, 

j = -OD 

G k>n :=» E (/"^(A:/n)) fc - 1 - J S(A ; /n)(/- / uS(A;/n))^ 1 - J , 

3=-co 
fc-1 

|«fc,n| < M/i 2 £(1 - 6fj,f<- k - 1 -fi (r> + ™ _/ ^ " J/ + M) 

<MLt(n + (nn)-P). 

For bounding i?fc >n , we have used (82), (84), a G A(/3, L) and then Lemma C.3. 
From (82) and (85), we have, for all fi G (0,/io], 1 < & < n, 

(94) |G fc , n |<M(l-<^) fe . 

From the previous bounds, we obtain, for all \i G (0,/xo] and 1 < fc < n, 

(95) |E[j£jg T ] - ^l n G k , n \ < + (vn)-P + (1 - 5^) k ). 
Now, by definition of G k ^ n we have 

(I - fiS(k/n))Gk,n(I ~ fjS(k/n)) + fxE(k/n) = G k , n , l<k<n, 

which gives, for all 1 < k < re, 

(£(tyn)(I - 2G Kn ) + 2G k>n )Z(k/n)) = 2fiZ(k/n)G k>n Z(k/n). 

From (94) sup 1<fc<n |Gfc jn | < oo uniformly over \i G (0,^0]. We thus have, for 
all n G (0, hq] and 1 < A; < n, 

|E(fc/n)(J - 2G M ) + (/ - 2G fe)n )£(fc/n)| < M/i. 

For any d x d matrix C and positive definite matrix S, the equation SB + 
BS = C has a unique solution linear in C and continuous in S over the set 
of positive definite matrices (see [16], Corollary 4.4.10). Hence, we obtain, 
for all fx G (0,//o] and 1 < k < n, \I — 2G kin ([/,)\ < Mfj,, which with (95) gives 
the claimed bound. □ 
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6.4. Proof of Theorem 6. We use the decomposition of 5 as 

£(u) + ( 5 (w) _ j(w,0)) + ^(v) _ j(v,0)j 

+ (j( w '°) - JW) + (jM) _ j(v)) + j(w) + j(v)_ 

Let ?? G (0,1). Applying (53), (69), (77), (89) and (90), there exists M > 
such that, for all t G [i], 1], fi G (0,//o] and n > 1, 

\\e n (t;n) - 0(f) - j^ ])n - J^ ln \\ 2At7 

< M^/m)^ + {{in)- 2 ? + /i). 

We then obtain (19) by applying Proposition 24 and Proposition 25, and 
using the fact that a is /3-Lipschitz to approximate a 2 ([tn]/n) by o~ 2 {t). 

6.5. Proof o/ Theorem 7. We use 5 = 6® +5^ + (5^ - j( w '°)) + ( j( w >°) - 
J(w)) + j(w)_ observe that there exists C > such that A(/3, L) C A(1,CL). 
Hence, we may apply (53), (59), (69) and (89), so that there exist M and 
fio such that, for all /i G (0,/ioL 

(96) sup sup \\e n (t;n)-e(t)- jg J Pj0jCT < M( ^ + (/m)^). 
(6,a)ec* te[n,i] 

Now, since /3 > 1, for all (#,cr) G C* and t G (0, 1], we may apply the Taylor 
expansion 0(u) = 6{t) + 0(v)(u — t), where v G [u,t], which yields \0(u) — 
6{t) + 6(t)(t-u)\ < \0(v)-6(t)\\u-t\ KLlt-uf' 1 . Hence (18) holds {(3 = 
1 and equal to the actual (3) at every point t > and we may apply 
Proposition 24 for computing J( w \ which easily yields the result. 

APPENDIX A 



Proof of Theorem 16. We first derive two simple lemmas valid under 
the assumptions of Theorem 16. We let vq and 5 denote some constants 
depending only on r, R\, ui, a\, Ci, A, B and s and we write for W^ k . 



Lemma A.l. For any a > 1, there exist 5 > and pq > such that, for 
all k G N and v G [0, uq] , 

k+r 

n {I-***) 



(97) 



<Ri)M 



i=k+l 



< e 



-<Si/ 



Proof. Under (C-l) we have \I - vA k \ < 1 for all fceN and G [0, i^i] 
(see Lemma C.l) so that we may assume a = 1 without loss of generality. 
We write 

k+r 

(98) [] (I-vAi) = I-vD k + S k , 

l=k+l 
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where 

k+r r 

D k := A l and 5 fc :=5^(-l)V A k+ij ...A k+h . 
l=k+l j=2 l<ti<— <ij<r 

For (3 G 1), where s is defined in (C-3) and v > 0, denote 

B fe H := {|A fc+1 | < u-P, \A k+r \ < u~ p } 

and B£(i/) its complementary set. From (C-l), we have that, for all v G 
(0,z/i], |i/£) fc | < ri/ 1- 0I(B fc (i/)) + rI(B£(i/)). Choosing i/ 2 G (0,z/i] such that 

rv\~ P < 1, we get that, for all i/ G [0,i/ 2 ], 2|i/Dfc|I(|i/D*| > 1) < 2rI(B£(i/)). 
Hence, using (98) and Lemma C.l, we obtain, for all v G [0,1/2], 



k+r 

n 

;=fc+i 



< 1 - ^A min ( J D fc ) + 2rI(B£(i/)) + \S k \. 



Equation (97) easily follows from this bound with (C-2) and the two follow- 
ing inequalities, which will be shown to hold for all v G [0, v\\: 

(99) < fl!)E fc [I(Bg(i/))] < O^, 

(100) I(0 fc < fii)E fe [|5 fe |] < MCj sA2)/ V sA2 , 

where M is some constant depending only on r. We now conclude the proof 
by showing these two last inequalities successively. 

Using the Markov inequality, we obtain E k [I(B%(v))] < Ef=fc+i A| > 
„-/»} < u s ^J2ii k+ i^k[\Ai\ s ], which implies (99) using (C-3). 

For all j = 2, . . . , r, for all ordered j-tuples 1 < i\ < ■ ■ ■ < ij < r, us- 
ing (C-l), we have, for all v G [0 } z/i], v j \A k+h ...A k+ij \ < v 2 \A k+il A k+h \. 
Hence, for some constant Mi depending only on r, for all v G [0, vi], 

E k [\S k \]<M lV 2 sup E k [\A k+i A k+j \]. 

l<i<j<r 

Put s = s A 2. The Holder inequality gives 

E k [\A k+i A k+j \] < {E k [\ A.+,| S ]} 1/i {IE fe [|^ + ,r 7(S ~ 1) ]} ( "" 1)/S - 

Observing that — s > and using (C-l), we have, for all v G [0, u\], 
EfcOAfc+jfrt*- 1 )] <E fc [|^ +i | s >- s V(s-i)+^ showing 

EfcOS-fcO^Mii/ 5 / sup E fc p fc+J n) 1/S ( sup E fe [|A fc+i | s ]V S ^ 

Ll<j<r J U<j<r J 

<Mii/ s sup E fe [|A fe+i | s ]<M lZ / s ( sup E fc [|A fe+i | s ] 

KKr \l<i<r 
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The proof of (100) then follows by bounding the above sup by a sum and 
by applying (C-3). □ 

Define N n := Y^Jq l(0lr < Ri)- We have the following. 

Lemma A. 2. There exist ao > and 7 > such that, for all n > 0, 

K [e- aoNn ]<e-^ nao V . 

PROOF. Observe that, using (C-4), for all fceN, 

®kr[V {k+1)r ] < VkrlXLtfkr > Rl) + BI(<j) kr < i?l)] 

= v fcr A I ^ fer>fll ^i? I ^ fcr - i?l) . 
Let {Wkr, k € N} be the process defined by 



Wq:=Vq and W kr 



\\N ( k-l)r 



D 



Vi 



kr ■ 



k> 1. 



Since N kr is .F/cr-measurable, we obtain, for all fcGN, 



Efcr[l^(fc+l)r 



— J Efe r [V(jt + i) r ] 



< 



] \ fe / A \ N kr-I(4>kr<Rl) 

aJ uj 



Hence, by induction Eo[Wfc r ] < Wq = Vq and, since 14 > 1, we get 



E f 



\\Nkr 

B 



< A fc+1 E C 



k+l 



V(ifc+l)r 



<A fc+1 E [W (fe+1)r ]<A fc+1 y . 
Noting that N kr +q = N kr for all g = 0, 1 . . . , r — 1, the proof follows. □ 

We now turn back to the proof of Theorem 16. From (C-l), for all fceN 
and v G [0, u%], 

(fc+l)r 

n 

Z=fcr+1 



(101) 



< e -(a/2)fl(0*r<fli). 



(fc+l)r 

n 

i=fer+l 
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where 5 is defined in Lemma A.l. Let n = mr + t, where m £ N and t = 
0, 1, . . . , r — 1. Equation (101) and the Cauchy-Schwarz inequality show that 

i p 



, 1/2 1/2 



where wi = Eq [e' 



-SuN n l 



and 



7T 2 =E 



'm— 1 

n< 

.k=Q 



(k+l)r 

n 

Z=fcr+1 



2p 



e tf "I(0 fcP <12i)+I(0 fcP >12i) 



Let J7o = 1 and recursively define Uk+i for k = 0,1,. . . 



+i 



(fc+l)r 



l=kr+l 



2p 



e & n^ kr <R 1 ) + l{ct>kr>Ri)}Uk. 



Applying Lemma A.l with a = 2p, we obtain that {Uk,J~kr) is a super- 
martingale. Consequently, 7T2 < 1. Lemma A. 2 and Jensen's inequality show 
that, for all v £ [0, ao/5], 

Eo [ e -^iV„j < Qfy[ e -«o-N»])&'/ao < e -^ n T/ , 
which concludes the proof. □ 



APPENDIX B 

Burkholder inequalities for the TVAR process. Throughout this sec- 
tion we let (3 £ (0, 1], L > 0, < p < 1, < c_ < <7 + and we set C := 
C(P, L, p,a-,a+). We further let r £ (p, 1). The following lemma is adapted 
from [9], Proposition 4. 



Lemma B.l. Let (H,f,P,{jf n ;r!eN}) be a filtered space. Let p > 2, 
PI1P2 G [L°o] swc/i i/iai p^ 1 +P2 1 = 2p~ x and Zef {Z n ;n £ N} &e an adapted 
sequence such that K[Z n ] = and ||^ n ||p < °o /or a// n £ N. T/ien 

n / n n \ 1/2 

(102) VZ< < 2pV||^|| Pl V||EZ^|| P2 . 



Proposition B.2. Assume (Al) u>i£/i a > 2 and let p > 6e suc/i i/iai 
2(p + 1) < q. Then there exists M > suc/i i/tai, for all (0,a) £ C, 1 < s < 
i<n and sequences {4>i} s <i<t in Li(M d , 1, M.;p), 

2 * 



i(X i;re ) -E6» iCT [</>i(X iin )]) 



<M sup |^| L i( P )E 

g/(p+l),e,£r i6{a,...,t} i=s 



9 «lLi(p)- 
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Proof. Let us apply Lemma B.l with Zj = <^>j(Xj jn ) — Ee i0 -[(^>j(Xj in )] 
and pi=P2 = q/(p + !)■ Prom the L q stability, we see that ||Zi|| g /(p+i),0 )(T < 

jr. 

M\<pi\ U ( p y It now remains to bound [^fc] ll^/(p+i),0,o-- Using the ex- 

ponential stability and the L q stability, Proposition 11 shows that, for all 
s < i < k < t, (6, a) G C, \\K]a\ Z k]\\ q /{p+i),e,a < Mr fc -%| Li(p) . The proof 
follows. □ 

Another application of Lemma B.l is the following result. 



Proposition B.3. Assume that q>5 and let p, r > be such that 
u := 2{p + r) + 5 < q. There exists a constant M such that, for all (9, a) € C, 
1 < s < t < n and sequences {li,j}s<i<j<t and {4>i}s<i<t, respectively in 
Li(R,l,M;p) and Li(R,l,R;r), 



2J Ji,j(X-i, n )o-i + i, n £i+l,n{<t>j{Xj in ) — Eg j(T [<f>j(Xj jn )]) 
s<i<j<t 2q/u,6,a 



(103) <Mi sup hi,j\hi(p) /Z l^lli(r) 
[s<i<j<t i=g+1 

/ t \ V2 / j \ 1/2 n 

+ SUp \(f>i\ U (r) WLi(r) SU P Hl7ijlLi(p) >■ 

\s<i<t i=8+1 / s<j<i\ i=s / J 

Proof. Let Ci,j : = 7i-i,j0^i-i,n)^i,n£i,n and C/j := 3 -(Xj )n ) - 
^^[^(Xj^)] for all s < i < j < t. For all s < i < j < t, Ui and £jj are 
./^-measurable. Throughout the proof, we denote by M some constant 
independent of s, t, n and (0,o~) G C. From uniform L q stability, for all 

s <i<j <t, 

(!04) \\Ci,j\\q/( P +i),e,a < M\ji- ld \ u(p) and \\Uj\\ q/ ( r+1):0iO . < M|<^| L i (r ). 
Denote Zj := Uj J2l= s +i Ci,j- The LHS of (103) then reads 



(105) 



s<i<j<t 



2q/u,0,a 



s<i<j<t 



J2 (Z 3 -E e ,AZj]) 

j=8+l 



2q/u,6,cr 



By Proposition 11, for all s < i < j < t, 

(106) me:*lUM q /(r+i),e,a < Mr^%| Li(r) . 
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Using (104) and (106), for all s < i < j < t, 

\EeAQ,jU J }\ = \EoAQX^[U j }]\ 

<\\CiAkeA\K::p \h^ 

< Mr J ~*|7i_ij| L i(p)|0j|Li(r)- 
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It then follows that 



E {KeACiMl^M-yl^ E l^'lLi(r) E ^"^^oo^l, 
s<i<j<t j=s+l i=s+l 

where 7^ := sup a < i<? -< t \%j\u(p) and A ■= E*= s +i \4>i\u{r); applying this 
bound to the RHS of (105), we obtain the first term of (103). 

We now bound the second term in the RHS of (105). Applying Lemma B.l 
with p = q/u, pi = q/(p + r + 2) and p2 = q/(p + r + 3), this terms satisfies 

E -Zj 

j=s+l 2q/u,6,a 

/ 4 t _ t ^ \V2 

-(~ E W Z j\\q/(p+r+2),0,aY H E ejT[^]ll?/(p+r+3),0,a- I 

where Zj := Zj — Kg a [Zi\. Applying the Hlder inequality, (104) and the 
Burkholder inequality (see [14], Theorem 2.12) shows that, for all s < j <t, 

\\ Z j\\q/(p+r+2),e,a < M\<f)j |Li(r)72,oo 

with 

3 1 V2 



72,oo 



sup 

s<j<t 



E I'y*— ijIliCp) 



i=s+l 



From the two last displays, we see that, in order to obtain the second term 
in the RHS of (103) and thus conclude the proof, it is now sufficient to show 
that, for all s < j <t, 



.00 ' 
k=j 

where 0^ := sup s<i<t 1 0i | Li(r) - I n i&ct, below we bound the LHS of the pre- 
vious equation by Aj + Bj and show this inequality successively for Aj and 
Bj. Denoting Ci,kU k := Ci,*A - E 0t<7 [Q jk U k ], we have 
t 

E ll E ofcr[' Z fc] llg/(p+r+3),0,<r < A j + B h 
k=j 
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E E £ic^t4] 



A* ■= E 

k=j i=s+l q/(p+r+3),0,a 
t k 

S J := E E H^O.ct" Ki,kUk] Wq/(p+r+3),0,cT- 
k=ji=j+l 



The bound on Aj is obtained as follows. The centering term in the definition 
in Aj may be forgotten by multiplying the leading term by a factor 2. Then 
we use the fact that £«,fc S Fj,n for all i < j and all k > i with the Holder 
inequality, and finally apply (106) and (104) with the Burkholder inequality 
for martingale sequences to the obtained norms. These three steps read 



k=j 



E <£%,*tfk] 

i=a+l 



g/(p+r+2),0,<x 



<2Ell4^[^]ll g /(r+l),^ 
fe=j 



E 

t=a+l 



<mE 



'felLi(r) 



q/(p+l),0,a 
3 \ V2 

E l7i-l,fcli,i(p) ) 

i=s+l / 



It remains to show a similar inequality for Bj. From (104) and (106), for all 

s<i<j<k<t, 



l^e'a [Cj,kUk] \\q/( p +r+2) ! 0,a 



< 



l^e'a [Cj,kUk]\\q/(p+r+2) 



0,cr 



(107) 



< 2 1 1 (j,fe E ^r [Uk] 1 1 q/{p+r+2),6,a 

< 2||Cj,/c||g/(p+l),0, CT |l E eJ ; o - n [ C/ fc]llg/(r+l),0,. 



ITj— l,fc|Li(p) l^fclLi(r-) ■ 

This bound is in fact useful only when k — j is large. We now derive another 
bound for the same quantity useful when j — i is large. Since for all i < j < k, 
E e jCj,k] = e£"M = 0, we have 



^e]a\Cj,kUk] = ^]a[4>k(^k,n)lj-l,k{^j-l,n)Oj,n£j,n] 

- ^e,a[4>kV^k,n)lj-l,k(^i-l,n)vj,n£j,n\- 
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Note that crj, n £j,n = Xj,n ~ 0j-i,n X j-i,n is linear in (Xj_i jn , X Jjn ). By 
Lemma C.2 and since sup^ n |0j-i, n | < oo, 7j_i j jfc(X J -_i in )0fe(Xfc 
a mapping of (Xj_i n , Xj n , Xfc n ), belongs to Li(M rf , 3, M;p + r + 2) and its 
norm is bounded by M |7j-i,fe|Li(p)l</ ) fc|Li(r)- Hence, applying Proposition 11 
and L 9 stability to the RHS of the previous display gives, for all £ < j < k, 

\\^]a[Cj,kUk]\\q/(p+r+?,),0,a < Mr j ~~ l \jj-l,k | Li(p) 1 4>k \hi{r) ■ 

Combining with (107), we get 

Applying these bounds to the definition of Bi, since for alH < t 

r^M fc ^)<2/(T(l-r)(l-v^)), 

i<j<k<t 

we finally obtain Bi < ^(f)}^ < oo$L> which yields the proof. □ 

APPENDIX C: TECHNICAL LEMMAS 

Lemma C.l. Let A be a 'positive semi-definite symmetric matrix and 
let I denote the identity matrix with the same size as A. Then \ I — A\ < 
1- \ m nM) + 2\A\I(\A\>l). 

Proof. Since | • | denotes the quadratic operator norm, we have \I — A\ = 
max(l - A min (A), A max (yl) - 1). If 1 - A min (A) > A max (,4) - 1, the claimed 
inequality is trivially true. Since A is positive semi-definite, A max (j4) = \ A\. 
If 1 — A m i n (^4) < A max (^4) — 1, \I — A\ = \ A\ — 1. In addition, in this case, we 
necessarily have \ A\ > 1. Hence, the right-hand side of the claimed inequality 
in this case reads 1 - A min (A) + 2| A\ = 1 + \A\ + A max (yl) - A min (yl) > \ A\ - 1. 
□ 

Lemma C.2. Let (E, | • |e) and (F, | • |f) be two normed spaces. 

(1) Let (G, | • |g) be a normed space. For any pi,p2 > 0, there exists C > 
such that, for all (j) £ Li(G, 1, F;pi) and ip G Li(E, m, G;p2), 

\<t> ° Mu{ PlP2 + Pl +p 2 ) < C|^lLi( Pl )(l + \Mut P \))- 

(2) Let (G, | • |g) be a normed algebra. For any pi,p2 > and any integers 
mi, m-2 > 1, there exists C > such that, for all 4> £ Li(E,mi, G;pi) and 
■ip G Li(F,m 2 ,G;p 2 ), 

\H\u( Pl +p 2 +l) < C|<^|Li(pi)|^|Li(p 2 )- 
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Lemma C.3. Let (5 > and v E (0, 1). Then there exist constants C\, C2 
depending only on j3 such that 

supt^l-u) 1 <du~^, 
t>o 

00 

8=1 

with the convention 0° = 1. Assume now that (J > 0. Then, as v \ 0, 

(108) X> - + ^ - s/3 ) = + ^""C 1 + OM). 

where Y is the Gamma function. 

Proof. The result is trivial for j3 = 0, so we assume /3 > 0. A straight- 
forward computation shows that sup t>0 i^(l — v) 1 is attained at t = to := 
— /3/log(l — z/). Since log(l — i>) is bounded above by — u, the first bound is 
obtained. The second bound is obtained by bounding (1 — v) s s^ by this sup 
for s < to + 1 and by bounding the remainder of the sum (whose terms are 
decreasing) by 

/ (l-u) s s^ds = T — — ™/ e- s s^ds, 

Jo (-log(l - v))p+ v Jo 

which easily yields the first bound. For all (5 > and v £ (0, 1), 

00 00 rs-\-l 

5:=^(l-i/) s ((s + l) /3 -s /3 )=/3^(l-i/) s / t p - l dt. 

s=0 s=0 s 

The proof of (108) follows by then writing 

roc 

(l-v)S<p / (l-vft^dt 
Jo 

= r(j3 + l)(-]og(l-v))- p <S. □ 
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